python-ecosys/debugpy: Add VS Code debugging support for MicroPython. - #1022

For those interested in AI coding, this was 95% written by Claude Code (Opus and Sonnet 4) as mentioned in the attributions above. I started prompting the build of this just this morning at 5am from my armchair with my infant asleep in my arms, using Termux on my phone to ssh into my linux box.

I had a clone of micropython with my historical work on getting pdb checked out (micropython/micropython#8767 and #499)

I also had a copy of the official cpython debugpy package checked out, this is the package used behind the scenes to drive python debugging in VSCode and similar IDE's. From past reviews I knew debugpy relies on threads, the old pydevd network debug engine as well a large RPC server which runs in a thread.

I figured some of this could be re-implemented if needed, or replaced with other servers already running on micropython :-)

So I started with a claude session in the debugpy folder and performed an initial repo analysis /init before asking:

perform detailed analysis  of the behaviour of this starting with listen mode.  write detailed specifications of the network API exposed and the data formats used there. then  document the pathways taken to get from network API down to the systrace api

Which produced DEBUGPY_ARCHITECTURE_ANALYSIS.md

I then kicked off : [corona@Telie micropython]$ claude --add-dir ~/debugpy/
and gave it this to kick off:

╭──────────────────────────────────────────────────────╮
│ > new  feature task; adding remote python debugging  │
│   support to micropython. I want to attach a         │
│   debugging session in vscode to a micropython       │
│   instance via debugpy.  we have a copy of cpython   │
│   debugpy to reference in ~/debugpy with an          │
│   analysis of its architecture in                    │
│   ~/debugpy/DEBUGPY_ARCHITECTURE_ANALYSIS.md I want  │
│   a minimal port of this library to micropython,     │
│   implementing the network listening interface       │
│   through to the pdb/systrace debug layer.there is   │
│   the micropython pdb implementation available at    │
│   ./lib/micropython-lib/python-stdlib/pdb/pdb.py     │
│   and any other dependencies should searched for     │
│   under ./lib/micropython-lib/ otherwise they might  │
│   need to also be implemented. the new debugpy       │
│   implementation can copy any of the cpython one     │
│   that makes sense, though for micropython smaller   │
│   is better. it should be built in a new folder:     │
│   ./lib/micropython-lib/python-ecosys/debugpy and    │
│   can be tested with the coverage variant of the     │
│   Unix port: /home/corona/micropython4/ports/unix/b  │
│   uild-coverage/micropython/home/corona/micropython  │
│   4/ports/unix/build-coverage/micropython            │
╰──────────────────────────────────────────────────────╯

WIthin just 1 hour of armchair vibe coding I had an initial implementation ready to test, along with test scripts and a written plan.

Around 10 am I was at my desk and had finished my morning meetings, so started testing it in the background while working on my other "real" projects.

It tooks quite a few iterations of testing in vscode for Claude to finish its implementation plan, adding features as it ran test scripts with me hitting the vscode "debug" button in between. Most of these tests failed badly in many different ways, enough that I was quite pessimistic at times because it really looked like it wasn't going to work ..... however I was still able to get other solid work done though during this time (which coincidentally was also using Claude Code; I've had 4 sessions actively on the go today) so I gave Claude a few chances to get it all going after a number of wrong paths were backtracked.

After all that though this screen capture was at 2:12 pm (and yes I ate lunch during that time too):

It took me a while to realise while reviewing afterwards and cleaning up the git tree that it hasn't actually pulled in pdb as a dependency, but re-implemented a simple version of everything needed but not much more; it's a rather minimal implementation!

Thanks Andrew,
very nice to see.
It took me more than a few attempts to get this running, but very rewarding.
Perhaps rebasing both PRs to master , and some tweaks fix mismatches in the names of the samples to will help there.
Its good to be able to do a even a simple step -by-step , even though there is now even more to be desired.

Thanks @Josverl good to hear it either for you, I still could hardly believe it worked for me!

I'd be interested to hear any notes about what was confusing / difficult to get going to feed into docs.

I assume some of it was getting paths right to import stuff? And/or compiling with the other features needed? Aka things that'll be better once finished and merged...

I will do some testing on hardware too, ensure that does work and document how to get it going.

I did think the branches were pretty well rebased up to date, I'll double check.

Oh yeah I'll eventually look into getting a useful representation of locals too, even if they end up basically just showing the array of values without names as per the current internal representation.

I'd be interested to hear any notes about what was confusing / difficult to get going to feed into docs.

I first got in a tangle by

• checking out the micropython repo on PR#8767
• then checking out ./lib/micropython-lib on this PR

blocked most attempts at building as make submodules breaks
After reverting ./lib/micropython-lib, and cloning this PR to a separate folder things got better

Building a firmware with "the updates to settrace in micropython/micropython#8767"
I still ran into build problems , essentially by a double definition of MICROPY_PY_SYS_SETTRACE .
In the readme of this PR it is part of the instructions , while in the "settrace PR" it is unconditionally defined. These two clash.

Matching up the paths vscode / remote paths, was not to difficult.

Open questions/ more play time needed :

• I am confused by the read-only source used by the debugger to step though. I can see it is retrieved from the remote, but not sure why this is needed , or how to configure it.
• why JustMycode : false is needed
• how debugging would work with a .mpy on the remote
• How to Enable VS Code's built-in DAP logging: ( I do not have these settings in my VSCode)
• Why the DAP_Monitor does not terminate or reset itself when the remote target exits

📚 likely relevant : What is the Debug Adapter Protocol?

@andrewleech
I have made some more notes after some additional testing I have done.
I have put them in a Gist notes.md for now as I do not want to drop too many comments here.
Would it make sense to start a Discussion on this topic , or do you prefer everything here ?

I'd be interested to hear any notes about what was confusing / difficult to get going to feed into docs.

I first got in a tangle by

• checking out the micropython repo on PR#8767
• then checking out ./lib/micropython-lib on this PR

blocked most attempts at building as make submodules breaks
After reverting ./lib/micropython-lib, and cloning this PR to a separate folder things got better

Yes the automatic make submodules in mpbuild pretty much breaks development within micropython-lib, I haven't figured out any way to resolve this without a newer/smarter algorithm in the make submodules target (which I haven't written yet)

Building a firmware with "the updates to settrace in micropython/micropython#8767"
I still ran into build problems , essentially by a double definition of MICROPY_PY_SYS_SETTRACE .
In the readme of this PR it is part of the instructions , while in the "settrace PR" it is unconditionally defined. These two clash.

Ah yeah, the readme in micropython-lib doesn't necessarily know about the updates in the micropython PR,

Matching up the paths vscode / remote paths, was not to difficult.

Open questions/ more play time needed :

• I am confused by the read-only source used by the debugger to step though. I can see it is retrieved from the remote, but not sure why this is needed , or how to configure it.

I haven't figured out the read only src either, it's annoying though. I think it's related to path matching issue between local and remote copies.

• why JustMycode : false is needed

Not sure if it is really needed?

• how debugging would work with a .mpy on the remote

The path matching would need to be smart enough to match PC side py to remote module (should be possible) and the mpy would need to have been made with opt=0 otherwise line numbers are stripped.

• How to Enable VS Code's built-in DAP logging: ( I do not have these settings in my VSCode)

Ah yeah I didn't actually test that!

• Why the DAP_Monitor does not terminate or reset itself when the remote target exits

Yeah that would be helpful, I'll add it

📚 likely relevant : What is the Debug Adapter Protocol?

Definitely a good reference to add to the docs thanks

Why the DAP_Monitor does not terminate or reset itself when the remote target exits

Yeah that would be helpful, I'll add it

I did some work to terminate on disconnect
See the PR to your fork

I was able to add (a first rough draft) of support for frame.f_locals on top of the pdb_support branch.

I was able to add (a first rough draft) of support for frame.f_locals on top of the pdb_support branch.

Oh wow awesome, I wanted to tackle basically exactly that! Looks great :-D
I was only just about to test my latest changes to this micropython-lib branch; hopefully the read-only window no longer pops up...

Now also able to resolve the names of local variables, at the cost of some memory per frame, and a change in the compiler.

Still a lot of checking and cleanup to do in that part of the code though

Pushed an update folding in the enhancement lineage that had been developed alongside this branch, plus several fixes found while building tooling on top of it. Fast-forward, no history rewritten.

Two are correctness fixes that affect anyone using this today:

Nested message pumps deadlocked the session. process_pending_messages() set a 1 ms socket timeout and restored blocking mode in its finally. The trace function calls it on entry to every new frame, so handling a message re-enters it, and the inner call's finally put the socket back into blocking mode underneath the outer pump. That pump's next recv() then waited for a message the client won't send until it has seen an event the pump itself produces. When it lost the race the session hung before wait_for_client() returned, so the target never ran and no stopped event was ever produced. It also cost ~100 ms per traced statement when it didn't deadlock. The nesting is now tracked rather than the timeout saved and restored, since MicroPython sockets have no gettimeout().

listen() now returns the bound endpoint before accepting. It previously bound, blocked in accept(), and handled initialize before returning, so a caller could only learn the endpoint after a client had already connected to it — unusable for anything that has to read the address in order to attach. accept() and the initialize handshake move into wait_for_client(), matching CPython debugpy, where listen() reports the endpoint and wait_for_client() blocks. Relatedly, port=0 now raises where getsockname() can't report the assigned port instead of silently substituting the default and advertising an address nothing is bound to.

Also included: DAP messages split across recv() calls are reassembled; a runtime capability probe (settrace/save_names/set_local/f_back) so tooling can ask what a firmware actually supports rather than inferring from a build name; a read-only hint on locals where the firmware can't write them back; evaluate honouring its context field so statements work from the Debug Console; and the pause / set-variable / local-modification work from the parallel lineage.

Exercised by a host-side DAP harness (a fake VS Code client driving real sessions against a built unix firmware): 278 passed. Happy to split any of this out if you'd rather review it separately.